Motorized robotic walker guided by an image processing system for human walking assistance

ABSTRACT

A motorized robotic walker is capable of moving automatically with the user through an algorithmic process using a 3D camera image processing system. The image processing system can measure relative motion of the user versus the robotic walker and a microprocessor can generate PWM signal to drive motors of the robotic walker so that the robotic walker can follow the user&#39;s motion automatically and provide assistance if needed.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of Application Ser. No.62/649,272, filed Mar. 28, 2018, which is pending and is herebyincorporated by this reference in its entirety.

ACKNOWLEDGEMENTS

This invention was made with government support under Grant No. R01NR016151 awarded by the National Institutes of Health. The governmenthas certain rights in the invention.

BACKGROUND Field of the Invention

Embodiments of the present invention relate to a mobility assistancedevice, specifically a motorized robotic walker guided by an imageprocessing system.

Background

Impaired mobility is ranked to be one of the most important factors thathave both physical and mental impacts on the patients' life. The impactsare especially serious for the rapidly expanding elderly population inthe United States, which is expected to reach 71 million, approximately20% of the U.S. population, by 2030. Existing assistive tools, such as acane or a walker/rollator, are helpful for such mobility-challengedindividuals by providing additional support in walking. However, suchtools also disrupt the users' walking rhythm and increase theirmetabolic energy consumption. Wheelchairs, especially poweredwheelchairs, are also used extensively among this population. Althoughwheelchairs are effective in transporting patients, they largelypreclude the users' lower limb muscle activities and bone load-carryingand accelerate the musculoskeletal degeneration of the user's lowerlimb.

To address the issues with existing assistive tools, a new motorizedrobotic walker for the mobility-challenged users is described herein.Unlike similar robotic walkers in prior works, no wearable sensors arerequired for the user. Impaired mobility has both physical and mentalimpacts on the patients' life. Impacts are especially serious for therapidly expanding elderly population in the USA. The problem alsohappens on handicapped people (e.g., stroke and SCI). Conventionalwalker/rollators are usually passive and, require user to pick up andpush walker forward. Walker/rollator disrupts the users' walking rhythmand increases their energy consumption. Wheelchairs largely preclude theusers' lower limb muscle activities and accelerate the musculoskeletaldegeneration of user's lower limb

BRIEF SUMMARY OF THE DISCLOSURE

Accordingly, the present invention is directed to a motorized roboticwalker for providing non-contact measurement of a user that obviates oneor more of the problems due to limitations and disadvantages of therelated art.

In accordance with the purpose(s) of this invention, as embodied andbroadly described herein, this invention, in one aspect, relates to amotorized robotic walker for providing non-contact measurement of a userto maintain a constant position relative to the user, includes a frame;at least two wheels disposed at a lower portion of the frame, each ofthe two wheels operatively connected to a motor; a 3D camera mounted ona portion of the frame, the camera capable of capture three-dimensionalorientation and position information about the user; a processor forgenerating the wheel rotation angles in real time; and motor drivers forcontrolling a speed of the motor and direction of the at least twowheels to maintain relative position of the frame and the user based onthe relative position data.

In another aspect, the method of operating a robotic motorized walkerhaving a frame, at least two wheels disposed at a lower portion of theframe, each of the two wheels operatively connected to a motor, a cameramounted on a portion of the frame, the camera capable of capturethree-dimensional orientation and position information about the user, aprocessor for generating wheel rotation information and driving themotors to maintain a substantially constant relative distance from auser, includes detecting the user's relative position and orientationrelative to the walker using the 3D camera, conducting an inversekinematic calculation using the user's relative position and orientationto determine equivalent wheel rotation angles for moving the at leasttwo wheels; generating motor-control PWM signals for each of the motorsto move the wheels according to the inverse kinematic calculation tomaintain a substantially constant relative distance from a user whereinthe camera is capable of generating both a color image and depthinformation.

Additional advantages of the invention will be set forth in part in thedescription which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Theadvantages of the invention will be realized and attained by means ofthe elements and combinations particularly pointed out in the appendedclaims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory only and are not restrictive of the invention, as claimed.

Further embodiments, features, and advantages of the motorized roboticwalker, as well as the structure and operation of the variousembodiments of the motorized robotic walker, are described in detailbelow with reference to the accompanying drawings.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory only,and are not restrictive of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures, which are incorporated herein and form part ofthe specification, illustrate the motorized robotic walker. Togetherwith the description, the figures further serve to explain theprinciples of the motorized robotic walker described herein and therebyenable a person skilled in the pertinent art to make and use themotorized robotic walker.

FIG. 1 is an exemplary (prototype) robotic walker according toprinciples of described herein;

FIG. 2 illustrate data points extracted from an image captured by a 3Dcamera.

FIG. 3 illustrates a control system with a PI motion controller of themotorized robotic walker according to principles described herein.

FIG. 4 shows control performance of the robot with a PI motioncontroller of the prototype in a turning experiment according toprinciples described herein.

FIG. 5 illustrates a control system with filters, sway suppressionfunction, and fully PID motion controller of the motorized roboticwalker according to principles described herein;

FIG. 6 illustrates comparison of the orientation angles pre-suppressionversus post-suppression;

FIG. 7 shows measurement of the user's orientation: ground truth vscomputer vision system in an experiment according to principlesdescribed herein;

FIG. 8 shows Measurement of the user's fore-aft displacement: groundtruth vs computer vision system in an experiment according to principlesdescribed herein;

FIG. 9 shows human-robot relative position (i.e., fore-aft movementtracking error) in a straight-line walking experiment in an experimentwith the controller in FIG. 5 according to principles described herein;

FIG. 10 shows human-robot relative orientation (i.e. turning trackingerror) in a continuous turning experiment with the controller in FIG. 5according to principles described herein;

FIG. 11 shows Comparison of the robot orientation trajectories with andwithout sway suppression according to principles described herein;

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments of the roboticmotorized walker with reference to the accompanying figures, in whichlike reference numerals indicate like elements.

An exemplary (prototype) robotic walker according to principlesdescribed herein is shown in FIG. 1. The exemplary walker is beneficialbecause it is intended to 1) accompany the user in walking with its ownpower and maneuverability, and 2) provides safety protection and poweredassistance (i.e., carrying the user forward) when needed. Additionally,the walker should be as light and simple as possible to facilitate itsfuture use in patients' daily life.

With the objective of assisting the users' ambulation in a safe andconvenient way, the robotic walker features two independently controlledwheels for the maneuverability of the robot, and two parallel bars forthe user support in walking. A 3D computer vision system is used tomeasure the relative position of the user versus the robot, and thecontrol commands are generated accordingly.

The prototype robotic walker (“robot”) shown in FIG. 1 includes acommercial framing system, which was utilized to construct a base 102 toexpedite the research. The base is not limited to this commercialframing system, but any appropriate base can be used. On the base 102,two parallel bars 104 were mounted using standard railing bars andhardware. Note that the back end of the robotic walker 100 is open,allowing the user 104 to enter and exit with no obstruction. With thisconfiguration, the user 104 can walk at the center of the openingdefined by a frame of the robotic walker 100, either freely withoutphysical contact with the robotic walker 100 or holding the parallelbars 104 when additional support and/or power assist is needed. Itshould be noted that the base or platform of a motorized robotic walkeras contemplated herein is not limited to the base used to construct theprototype. The platform, base or frame, may be of any feasibleconstruction and may be of a height appropriate to the user's needs. Theheight of the frame may be adjustable. In an aspect, the handle bars areof a height sufficient to catch a person/user who might fall and needthe support of the handle bars. The size of the frame is a trade offwith the maneuverability of the robot.

As described below, the camera should be located to capture theappropriate data points. Thus, the physical parameters of the walker canbe such that are tailored for the needs of classes of users so long asthe data points can be appropriately captured. It is contemplated thatthe camera can be mounted lower or pointed to a location lower than thetorso with appropriate data capture and algorithm implementation.

The frame may include a lightweight harness connected to the frame ofthe robot to provide stability to a user or keep the user from falling.

For the mobility of the robot, two independently controlled hub motors108 are mounted at the front of the robot, and two caster wheels aremounted at the rear of the robot, e.g., for balance. In anotherconfiguration, the hub motors could be located to the rear of the robotand the casters at the front. It is conceivable that the robot couldinclude four or more hub motors. In the presently-described prototype,the hub motors (280-1342M, Hub Motors, Rogers, Ariz.) are brushedpermanent magnet DC motors with 180 W continuous power rating. Poweroutput of the motors are regulated with two PWM servo drives (AZBDC20A8,Advanced Motion Controls, Camarillo, Calif.) (not shown), and onboardcontrol calculation is implemented with a microcontroller (not shown).According to principles described herein, the robot may move at avelocity faster than the user. The robot moves by virtue of the hubmotors, external battery 110, external control system and motor drive tocontrol the speed of the motor in the prototype. The robot may includemore than one battery. For example, batteries may be mounted adjacentthe wheels at the sides of the robot.

To provide noncontact measurement of the user 106 versus the robot 100,a 3D computer vision system is implemented. In the prototype, an imageis taken with a 3D camera 112 (Realsense R200, Intel, Santa Clara,Calif.) mounted at the front of the robot 100 through a mounting bracket114 with adjustable height and orientation. This camera 112 is able togenerate both color image and depth data frame at 640×480 resolution and60 fps. Each depth data frame provides a 3D point cloud P of the user'supper body bounded by 100 (red and green points as shown in the sampleimage in FIG. 2). The user's torso data (green points) is extracted bysegmenting the grey image obtained by projecting the point cloud data toan accumulator array on the horizontal plane. After detecting theLeast-square Plane of the torso surface and its normalvector^(n)={n_(x), n_(y), n_(z)}, the relative orientation between therobot and the human can be obtained. The relative position can beobtained with a similar approach. In the prototype, the trackingalgorithm is programed in an Up Board with Intel Atom processor runningUbuntu system, and a serial cable is used to transmit theposition/orientation data to the microcontroller for robot motioncontrol.

For the real-time motion control of the robot 100, the goal is tomaintain a constant relative position and a zero-degree orientationbetween the user and robot as much as practical. The relative positionΔy and orientation Δθ are measured with the 3D computer vision systemdescribed above, and inverse kinematic calculation is conducted todetermine the equivalent wheel rotation angles ϕL and ϕR (i.e., theangles the wheels need to turn to eliminate the position and orientationerrors). Subsequently, a proportional-integral (PI) controller isimplemented for the control of each wheel, generating the motor-controlPWM signals for the motor drives. For the implementation of the PIcontroller, a rotary magnetic encoder (AS5311, AMS, Premstaetten,Austria) is mounted on each wheel to provide real-time feedback of thewheel position. A block diagram of the control system is shown in FIG.3.

The prototype of the robotic walker was fabricated, and experiments wereconducted to demonstrate the functionality of the robot in two majoraspects. First, the 3D computer vision system was tested to measure thehuman orientation and position, and then the algorithm was simulated inthe RealSense software. As shown in FIG. 2, the algorithm successfullycalculated the orientation (θ=26.51°) by processing the 3D image shownin FIG. 2. In lieu of the 3D camera, laser positioning, such as LIDAR,could be used to provide positional information to allow the motorizedwalker to maintain appropriate distance from a user and to avoidobstacles, as discussed below. In some cases, laser positioning mightrequire the user to wear sensors or beacons to be sensed by the LIDAR.

The control performance is shown in FIG. 4, in which the robot wascommanded to turn right by 90°, stay for 2 seconds, and turn back to theoriginal orientation. As shown in FIG. 4, the prototype robot was ableto track the desired orientation change. Some delay in tracking responseexists, primarily due to the slip of the wheels on the ground, whichlimits the maximum gains that can be used in the PI controller.

In another implementation of the walker according to principlesdescribed herein, the motion controller includes an orientation signalpreprocessing module for sway suppression and a hybridproportional-integral-derivative (PID) controller with lower-passfiltered differentiation. A schematic of an exemplary motion controlleraccording to this implementation is shown in FIG. 5.

Sway suppression in which the negative effects of sway motion arereduced while the robot is still responsive to the user's turning motionare provided according to principles described herein. A suppressionband is defined based on the angular range of the sway motion:−Θ≤θ≤Θ  (2)

where θ is the measured upper body orientation and Θ is the width of thesuppression band, which can be determined through observation or tuning.Subsequently, a suppression function is defined, which meets thefollowing requirements within the suppression band: 1) continuous; 2)symmetric with respect to the origin; 3) derivative close to zero aroundthe origin; and 4) equal to the independent variable (θ) at the boundary(to ensure the overall continuity). A simple solution adopted in thiswork is a 3^(rd)-order polynomial, which provides sufficient flexibilityin tuning. The overall output function is defined as:

$\begin{matrix}{\theta^{\prime} = \left\{ \begin{matrix}{{a_{3}\theta^{3}} + {{{{sgn}(\theta)} \cdot a_{2}}\theta^{2}} + {a_{1}\theta}} & {{{when}\mspace{14mu} - \Theta} \leq \theta \leq \Theta} \\\theta & {else}\end{matrix} \right.} & (3)\end{matrix}$

where θ′ is the output orientation angle (post-suppression), anda₁/a₂/a₃ are the tunable parameters. An example comparison of theorientation angles pre-suppression versus post-suppression is shown inFIG. 6. The post-suppression orientation signal θ′, combined with therelative position in the longitude direction d, is used to calculate thedesired wheel rotation angles through inverse kinematic calculation.

On the lower level, a wheel motion controller was developed to regulatethe wheel rotation. A hybrid proportional-integral-derivative (PID)controller with lower-pass filtered differentiation is used.

A computer vision system according to principles described herein wasvalidated through a comparison with the measurement results obtainedwith the standard marker-based motion capture system (MCS).Subsequently, the performance of the system was quantified through theexperiments of simple human-initiated movements and navigation of anindoor environment tracking human movement.

To evaluate the performance of the computer vision system, an OptiTrackmotion capture system (MCS) was used to obtain the ground-truthmeasurement of the user's orientation and position. As shown in FIG. 7,the estimation error is shown as the dashed red line. As can be observedin this figure, the measurement of the computer vision system matchesthe MCS measurement very closely: the root mean square (RMS) of theerror is 2.16 degree and the maximum error is 4.14 degree. The resultsupports that the computer vision system is capable of computing thehuman orientation with small errors.

The second experiment was conducted to evaluate the accuracy of thecomputer vision system for position detection. A human subject stood inthe front of the camera about 1.3 m, then moved forward about 0.5 m andmoved backward to the starting point. His walking speed was about 0.13m/s. In this case, since the vertical and lateral displacements arenegligible, only the position estimation along the fore-aft axis wascompared. By setting the start point as the base point, thedisplacements of the human center estimated by the MCS and by the imageprocessing system were plotted in FIG. 8. It shows that the humansubject moves about 1 m in 8 seconds. The RMS of for the human centerdisplacement is 0.012 m and the maximum error is 0.024 m. The resultsupports that the image processing system is capable of computing thehuman position with small errors.

Based on the validation results presented in the section above, it isreasonable to assume that the relative position and orientation capturedby the computer vision system is sufficiently accurate for robot controlpurpose.

Experiments were conducted to evaluate the robot's performance intracking a human user's continuous walking and turning movements. First,the human user walked forward in a straight line at a speed ofapproximately 1.0 m/s (a typical walking speed for older adults). Thehuman-robot relative position (i.e., fore-aft movement tracking error)was recorded for 30 seconds, as shown in FIG. 9.

As can be seen in FIG. 9, the tracking error is very small compared withthe overall movement, with the maximum at 30 mm and the average at 10mm. The subject's feedback indicates that the controller was able tocreate a comfortable experience of the robot accompanying the user,despite the fact that the subject was constantly moving.

A similar experiment was conducted for the robot to track a human'scontinuous turning movement. During the experiment, a human user madecontinuous turning movement by following a circle at a speed ofapproximately 60 degree/s. The human-robot relative orientation (i.e.turning tracking error) is shown in FIG. 10. Similar to thestraight-line walking, the robot was able to change its orientation andeffectively tracking the user's turning movement.

Sway suppression, as described herein, may be a component of the robotcontroller. As such, a set of comparison experiments was conducted toevaluate its effectiveness in robot control. In the experiments, thetest subject walked in a straight line at a speed of approximately 1m/s, and the robot's sway movement is measured with and without the swaysuppression algorithm implemented. For such measurement, an inertiameasurement unit (MPU6050, InvenSense, San Jose, Calif.) was attached tothe robot frame to measure its absolute orientation change. FIG. 11shows the comparison of the robot orientation trajectories with andwithout sway suppression. It can be clearly observed that the magnitudeof the robot sway reduced significantly after the sway suppressionalgorithm was applied.

Accordingly, the motorized robotic walker according to principlesdescribed herein is capable of moving automatically with the userthrough an algorithmic process to provide the assistance if needed. Themotorized walker can provide walking assistance automatically if needed.It is the first time to use an image processing system to realizeautomatically navigation for a walker. Using automatic navigationreduces users' metabolic energy consumption.

Additionally, a light source can be used to allow for use in completelydark environments. The addition of a light source can assist with theimage capture and provide the user the ability to see. Alternatively,the infrared information in the camera could be used in darkenvironments to enhance image capture.

Obstacle and planar object detection using sparse 3D information for arobot will assist the user and the user's mobility. Two cameras may beinstalled in front of the robot for road map and obstacle detectionmaking use more practical by avoiding accidents with external objects.

Advantages expected according to principles of the present applicationinclude for users a positive health impact on users' physical and mentalconditions and easy operation not addressed by current walkers ormobility assistance devices. The robotic walker according to principlesdescribed herein provides a novel application of imaging processingguidance in medical device.

Various publications have attempted to address issues with users invarious ways, but none address the issues addressed by the presentinvention. Some background materials are provided in:

Mobile Walker, Monitoring and Information System (US20170354564A1) Themonitor and information system is to measure vital parameters containsfor instance heart frequency, blood pressure, blood oxygen content,blood sugar content and the ability of the user of the mobile walker torespond and move

Zwibel, H., 2009, “Contribution of Impaired Mobility and GeneralSymptoms to the Burden of Multiple Sclerosis.” Advances in Therapy, vol.26, no. 12, pp. 1043-1057.

The Merck Company Foundation, 2007, “The State of Aging and Health inAmerica 2007,” Whitehouse Station, N.J.: the Merck Company Foundation.

Lee, B., Ko, C., Ko, J., Kim, J. S., Lim, D., 2015, “Suggestion of NewConcept for Mobility Assistive System Based on Wheelchair Platform withGait Assistive Function Controlled by Artificial Pneumatic Muscle.”Biomedical Engineering Letter, vol. 5, no. 2, pp. 87-91.

Homich, A. J., Doerzbacher, M. A., Tschantz, E. L., Piazza, S. J.,Hills, E. C. and Moore, J. Z., 2015, “Minimizing Human Tracking Errorfor Robotic Rehabilitation Device.” ASME Journal of Medical Devices,vol. 9, no. 4, p. 041003.

Qian, X., Ye, C., 2014, “NCC-RANSAC: A Fast Plane Extraction Method for3D Range Data Segmentation,” IEEE Transactions on Cybernetics, 44 (12),pp. 2771-2783.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the present inventionwithout departing from the spirit or scope of the invention. Thus, it isintended that the present invention cover the modifications andvariations of this invention provided they come within the scope of theappended claims and their equivalents.

Throughout this application, various publications may have beenreferenced. The disclosures of these publications in their entiretiesare hereby incorporated by reference into this application in order tomore fully describe the state of the art to which this inventionpertains.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. It will be apparent to persons skilledin the relevant art that various changes in form and detail can be madetherein without departing from the spirit and scope of the presentinvention. Thus, the breadth and scope of the present invention shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A motorized robotic walker for providingnon-contact measurement of a user to maintain a constant positionrelative to the user, comprising: a frame; at least two wheels disposedat a lower portion of the frame, each of the two wheels operativelyconnected to a motor; a 3D camera mounted on a portion of the frame, thecamera capable of capturing three-dimensional orientation and positioninformation about the user; a processor for generating the wheelrotation angles in real time; and motor drivers for controlling a speedof the motor and direction of the at least two wheels to maintainrelative position of the frame and the user based on the relativeposition data.
 2. The motorized robotic walker of claim 1, wherein thecamera is capable of generating both color image and depth information.3. The motorized robotic walker of claim 2, wherein the depthinformation is depth frame data.
 4. The motorized robotic walker ofclaim 3, wherein the depth frame data is 3D point cloud P of a portionof the user's body.
 5. The motorized robotic walker of claim 4, whereinthe portion of the user's body is the user's upper body.
 6. Themotorized robotic walker of claim 1, wherein the processor generates PWMsignals for driving the motors.
 7. The motorized robotic walker of claim6, further comprising a sensor on each of the at least two wheels toprovide feedback to the processor of wheel position.
 8. A method ofoperating a robotic motorized walker having a frame, at least two wheelsdisposed at a lower portion of the frame, each of the two wheelsoperatively connected to a motor, a camera mounted on a portion of theframe, the camera capable of capturing three-dimensional orientation andposition information about the user, a processor for generating wheelrotation information and driving the motors to maintain a substantiallyconstant relative distance from a user, the method comprising: detectingthe user's relative position and orientation to the walker using the 3Dcamera, conducting an inverse kinematic calculation using the user'srelative position and orientation to determine equivalent wheel rotationangles for moving the at least two wheels; generating motor-control PWMsignals for each of the motors to move the wheels according to theinverse kinematic calculation to maintain a substantially constantrelative distance from a user; wherein the camera is capable ofgenerating both a color image and depth information.
 9. The method ofclaim 8, wherein the depth information is depth frame data.
 10. Themethod of claim 9, wherein the depth frame data is 3D point cloud P of aportion of the user's body.
 11. The method of claim 10, wherein theportion of the user's body is the user's upper body.
 12. The method ofclaim 11, wherein the processor generates PWM signals for the motor. 13.The method of claim 12, further comprising an encoder sensor on each ofthe at least two wheels to provide feedback to control the rotation ofthe wheels.